Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UIA objects: introduce a global suggestion list item class for Windows 10. re #6241 #6274

Merged
merged 31 commits into from
Jun 23, 2017

Conversation

josephsl
Copy link
Collaborator

Windows 10 uses suggestions list for vairous things, including Start menu suggestions, Store recommendations, Settings app and others. Thus introduce NVDAObjects.UIA.SuggestionListItem, which derives its power from searchui.py version (searchui.py version is gone).

Suggested what's new entry:

  • Category: not sure
    In Windows 10, you can use up and down arrow keys to review suggestion results (e.g. Settings app suggestions and Store recommendations).

Thanks.

@derekriemer
Copy link
Collaborator

Hi, maybe we should create a UIA behaviors package?

Sent from a mobile device.
Please disregard errors as this is a smaller device.

On Aug 16, 2016, at 21:25, Joseph Lee notifications@github.com wrote:

Windows 10 uses suggestions list for vairous things, including Start menu suggestions, Store recommendations, Settings app and others. Thus introduce NVDAObjects.UIA.SuggestionListItem, which derives its power from searchui.py version (searchui.py version is gone).

Suggested what's new entry:

Category: not sure In Windows 10, you can use up and down arrow keys to review suggestion results (e.g. Settings app suggestions and Store recommendations).
Thanks.

You can view, comment on, or merge this pull request online at:

#6274

Commit Summary

UIA objects: introduce a global suggestion list item class for Windows 10. re #6241
File Changes

M source/NVDAObjects/UIA/init.py (19)
M source/appModules/searchui.py (18)
Patch Links:

https://github.com/nvaccess/nvda/pull/6274.patch
https://github.com/nvaccess/nvda/pull/6274.diff

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@josephsl
Copy link
Collaborator Author

Hi,

UIA behaviors: I'm thinking not.

@josephsl
Copy link
Collaborator Author

@michaelDCurran, I'd like to request a review please. Thanks.

if self.UIAElement.cachedAutomationID in ("TextBox", "SearchTextBox"):
clsList.append(SearchField)
try:
if self.parent.UIAElement.cachedAutomationId.lower()=="suggestionslist":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be very costly and could end up being recursive -- I had made the same mistake with Microsoft Edge code earlier this year. Items within a suggestionList don't have a useful UIA className or automationID?

def event_UIA_controllerFor(self):
# Only useful if suggestions appear and disappear.
focus = api.getFocusObject()
if len(self.controllerFor)>0:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you'd want to check that self is focus before checking len(self.controllerFor)... You already fetched focus but didn't use it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I tested it with Settings and what not, just using self.controllerFor was enough, but I'd be happy to check the focus to really make sure (thanks for catching this).

if self.UIAElement.cachedAutomationID in ("TextBox", "SearchTextBox"):
clsList.append(SearchField)
try:
if self.parent.UIAElement.cachedAutomationId.lower()=="suggestionslist":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is absolutely necessary to check the parent, then you can use raw UIA calls. Using the UIAHandler.handler.baseTreeWalker and get the parent element for self.UIAElement and check its automationID.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I'll read EdgeRS1 branch to see what I can learn from there before I proceed. Thanks.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I'll read EdgeRS1 branch to see what I can learn from there before I proceed. Thanks.

@michaelDCurran
Copy link
Member

@josephsl: how did you go with removing the call to self.parent when detecting suggestionLists? If it is necessary still, then you should use raw UIA calls to stop possible NVDAObject recursion which could be a large performance hit.
Also, it would be good to abstract out the announcements of the opening and closing of suggestions into a behavior, perhaps with some kind of method or event each implementation can call.

@josephsl josephsl force-pushed the i6241UIASuggestionListItem branch from 913a112 to 04a1c0b Compare January 18, 2017 04:44
@josephsl
Copy link
Collaborator Author

Hi,

Behaviors: I agree - after thinking about it, suggestions detection display is something that other API's would like to participate in (not only Start search box in Windows 10, but also address bar in Firefox, Tell Me suggestions in Office 2016 and so on). Search Field in UIA (Windows 10) will now use this behavior.

Thanks.

@josephsl
Copy link
Collaborator Author

Hi,

Regarding using self.parent: yes, this is the case. I'll try your suggestion of using UIA calls directly. Thanks.

@derekriemer
Copy link
Collaborator

derekriemer commented Jan 18, 2017 via email

@josephsl
Copy link
Collaborator Author

Attached is a zip file used for announcing appearance/disappearance of auto-suggestions (credit: @derekriemer). Thanks.
nvda_i6241waves.zip

@michaelDCurran
Copy link
Member

A few thoughts:

  1. @jcsteh has suggested that if we have a sound, then it should only ever be a sound, and not to allow configuring for speech. Thus removes the need for the combo box in settings. I think I agree.
  2. I don't think it is necessary to deliberately stop it from occurring in the Start Screen. In fact if it is only a sound it does not change existing speech, and helps the user to be come more familiar with the sound.
  3. What about Microsoft Edge Address bar? I think its automationID is addressEditBox. However, I'm not sure what their suggestions list is or if it will match. But we'd definitely want to support it.

…s 10. re nvaccess#6241

Windows 10 uses suggestions list for vairous things, including Start menu suggestions, Store recommendations, Settings app and others. Thus introduce NVDAObjects.UIA.SuggestionListItem, which derives its power from searchui.py version (searchui.py version is gone).
Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.
Controller For property is used by an element to let another element depend on values given by the source element, useful if suggestions should be listed for a search field among other possibilities.
For now, Controller For property has been added to property to event names map in UIA Handler. A UIA object that handles this event (a search box) is next.
… and announce appearance of suggestions. re nvaccess#6241

Coming from Windows 10 App Essentials add-on: a Search Field is now available that'll detect controller for property event and announce either 'suggestions' or 'suggestions closed' if suggestions appear or disappear, respectively. This takes care of both Start Menu and other fields in Windows 10 (works best in Anniversary Update and later).
…roller for object. re nvaccess#6241

Review from Mick Curran (NV Access): Do not waste a function call for fetching focused element. Besides, make sure to check if the focused control is the search field before proceeding to announce appearance of suggestions.
…stions. re nvaccess#6241.

Reviewed by Mick Curran (NV Access): announcement of search suggestion is something that NVDA should handle for other API's, such as suggestions in Firefox address bar, search suggestions in universal apps and so on.
A new behavior named 'Suggestion' has been added that allows subclasses to provide custom routines when suggestions appear and disappear. This is handled by event_suggestionOpened and event_suggestionClosed, and by default NVDA will speak and braille this event.
…n to announce suggestion appearance. re nvaccess#6241.

Reviewed by Mick Curran (NV Access): Based on the new Suggestion behavior mix-in, it is now possible for various objects to provide custom routines to let users know the appearance of suggestions. Thus UIA/Search Field is the first object to use this routine.
…suggestion list (parent). re nvaccess#6241.

Reviewed by Mick Curran (NV Access): it is better to use raw UIA for obtaining parent element. However, one must be careful to catch COM and value errors (COM because the element might not be there, and value because NULL pointer access is logged). The raw UIA method was also recommended by Derek Riemer.
…ced in Windows 10 Start menu for consistency with earlier versions of Windows. re nvaccess#6241.

iN Windows 10 Start menu, suggestions are announced automaticlaly, so no need to provide suggestion announcements.
…tions' so the behavior can be better described. re nvaccess#6241.

If one inherits from 'Suggestion', the overall impression would be that the field is only going to display suggestions nad nothing else. Many suggestions are shown when text is entered, thus it is better to say 'EditableTextWithSuggestions' to better reflect what the behavior actually does.
Also, a sound will be played to let users know the appearance of suggestions. (default vlaue).
…configure how auto-suggestions should be announced. re nvaccess#6241
…ss#6241.

In the user guide, an explanatory text has been added to describe what auto-suggestions are.
Comments from Mick Curran and Jamie Teh (NV Access): no need for a separate message option when announcing appearance of auto-suggestions, as the sound cue will let the user of this fact. However, braille users should be notified of this regardless of this flag being turned on (deaf-blind users should be notified of this).
User guide and settings dialog: changed the control type and label for auto-suggestions setting to reflect change in behavior.
josephsl added 2 commits May 11, 2017 09:42
…lement fetcher fails. re nvaccess#6241.

In some cases, when Start menu opens, it isn't announced by NVDA. As a result, parent element fetcher fails when trying to instantiate suggestions list item, with a traceback that ends with UnboundLocalError. Catch this by moving the SuggestionsListItem selector to inside of the try block.
josephsl added 4 commits May 31, 2017 10:39
…tions close for consistency with other situations (such as browse mode toggle). re nvaccess#6241.
…sh messages. re nvaccess#6241.

Suggestion from Davy Kager: provide a way to let braille users read search suggestion items. This is done by emulating some parts of speech.SpeakObjectProperties except the name and position info map will be fetched (position info map fetching is contingent on whether report position info setting is enabled from Object Presentation dialog). Ideally, NVDA objects should have a way to fetch braille flash messages for controls.
Also reworded docstring for SuggestionListItem so it cna include other UIA-based suggestion list items such as Windows 8.x search results.
…lle. re nvaccess#6241, nvaccess#6414.

Instead of constructing the likely flash message, use a function used as part of Core issue 6414, which is much simpler than constructing the flash message from scratch.
michaelDCurran added a commit that referenced this pull request Jun 8, 2017
@jcsteh
Copy link
Contributor

jcsteh commented Jun 14, 2017

I could swear this was working the other day, but maybe I was running the wrong build. STR:

  1. Open the Start menu.
  2. Search for an app.
  3. Press the applications key.
  4. Use the up and down arrows to move through the context menu items.
    • Expected: NVDA should report items like Pin to Start, Pin to taskbar, etc.
    • Actual: It repeatedly reports what you typed into the search box and nothing more.

I think the UIA suggestion list item code needs to be extended to cover items with a parent automation id of "contextMenu".

@josephsl
Copy link
Collaborator Author

josephsl commented Jun 14, 2017 via email

@jcsteh
Copy link
Contributor

jcsteh commented Jun 15, 2017 via email

@josephsl
Copy link
Collaborator Author

josephsl commented Jun 15, 2017 via email

josephsl added 2 commits June 14, 2017 17:55
…ontext menu as a suggestions list item. re nvaccess#6241.

Oddly, the same behavior that's applied to suggestion items must work in Start suggestion's context menu, otherwise menu items will not be announced.
michaelDCurran added a commit that referenced this pull request Jun 16, 2017
@michaelDCurran
Copy link
Member

@josephsl: could you please merge master and resolve the conflict. It is just the addition of the docstring to NVDAObjects.UIA. Then I can merge this to master. Thanks.

@josephsl
Copy link
Collaborator Author

josephsl commented Jun 23, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants